Lecture 20: Spectral Methods

For simplicity we will focus on a simple Gaussian Mixture Model. Consider a mixture of k spherical gaussians in R which is the following generative process. Let w ∈ ∆([k]) denote a distribution and let μ1, . . . , μk ∈ R be the mean vectors. Each point xi is generated by first choosing a component hi ∼ w and then xi ∼ N (μhi , I). We are given n samples x1, . . . , xn drawn according to this process. Such models are called latent variable models since the component assignments hi are latent or unobserved. The standard approach of maximum likelihood estimation is typically intractable in a latent variable model due to this unobservability. Specifically, the negative log likelihood here is